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(54) DOCUMENT IMAGE REARRANGEMENT RUNG DEVICE 

(57)Abstract: 

PURPOSE: To efficiently execute a retrieval by extracting and 
separating automatically areas of different attributes such as a 
heading, a text, a chart, a photograph, etc., in a document image, 
and accumulating them at every item so that the retrieval can 
be executed by a suitable retrieving method at every area of 
the attributes. 

CONSTITUTION: An input document image inputted by scanning . 
a newspaper, etc., by a document image input means 1 is 
separated into each attribute area of a heading 3-1, a text 3-2, 
a chart 3-3, and a photograph 3-4. Subsequently, as for the 
areas of the heading 3-1 and the text 3-2. a character 
recognition processing is performed and they are converted to 
code data, and as for the areas of the chart 3-3 and the 
photograph 3-4, an image compression processing is performed, 
and thereafter, they are brought to filing to each separate 
attribute item. In the retrieving means 6, or the retrieving means 
7, the retrieval can be executed by selecting one of a key word 
retrieval to the heading 3-1 or the text 3-2, and a retrieval of a 
similar image to the chart 3-3 and the photograph 3-4. In such 
a way, the retrieval/display time for a document image is 
shortened, and also, by rearranging the document image, the 
data compression can be executed. 
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SPECIFICATION 

1. TITILE OF THE INVENTION 

DOCUMENT IMAGE RE -ARRANGEMENT FILING APPARATUS 
5 2. WHAT IS CLAIMED IS: 

A document image, re-arrangement filing apparatus 
comprising: 

document image input means for inputting document images- 
region -dividing means for automatically extracting from 
10 the document image input by said document . image • input means 
different attribute regions such as a headline region, a text 
region, a drawing region and a photograph region, and dividing 
them; 

itemized file storing means for accumulating and- storing 
15 items such as the headline, the text region, the drawing region 

and the photograph region, extracted and divided by the region 

dividing means 

retrieving means for retrieving data of the items stored 

in the itemized file storing means, by item; 
20 document image re-arrangement means for retrieving 

desired document image by retrieval of the retrieving means, 

and re-arranging the data in the items to output the entire 

document image, 

3. DETAILED DESCRIPTION OF THE INVENTION 
25 [FIELD OF THE INVENTION] 



The present invention relates to a document image 
re-arrangement filing apparatus capable of automatically 
extracting images in regions consisting of various attributes 
such as a headline, a text, a drawing, a photograph and the like, 
to store item by item, re-arranging each of the entire item in 
image such as a scrap of a newspaper. 
[PRIOR ART] 

It is proposed now for an apparatus that files document 
images such as a scrap of .a newspaper to- employ, a system, for 
automatically ' extracting the complicated layout shapes 
according to eac^h attribute ( a headline, a text, a drawing, a 
photograph) region of the document image, and re-arranging the 
image data coi^aitly. 

For example, thexe is proposed a database editing and 
controlling, ^gyifem that . Extracts image of a newspaper that is: 
input by an iMp^ ^ input apparatus , as a pattern with, use of a 
man-machine iilfe:f%ee% stores the image-data after structuring 
the- data Wy ^^^MinraMs^^0^±^±^q^ the data to reduce the data 




Engineer sU/^ pp. 65-7 2) . 

SimS^ffirly, an automatic layout system capable • of 
automatically extracting the document structure such as a 
hierarchical structure based on the concepts such as a title, 
a chapter and a paragraph., and a reference structure between 



a text and drawings by using format information of the document 
or keywords, and executing formatting or layout of the document 
with use of a format based on the document structure and knowledge 
about layout technique (JOHOSHORI ASSOCIATION "DOCUMENT 
5 PROCESSING AND HUMAN INTERFACE", 20-3, September 8, 1988) . 
[OBJECT TO BE SOLVED BY THE INVENTION] 

In order to obtain the desired document image with use 
of the above-mentioned techniques, however, the entire document 
image has to be retrieved and checked, which. is a so troublesome', 
10 job requiring a long period of time. Further , the image document .< 
contains various data such as a headline, a text, a drawing, 
a photograph and the like in a mixed manner, so that in some • 
cases, it turns out wasteful from a viewpoint of data storage 
system. 

15 The present invention intends'to provide a document image 

re-arrangement filing apparatus capable of reducing a period 
of time required for retrieving and displaying the document image, 
and compressing data by re-arranging the. document image data 
to remove redundant space, in the document. 

20 [MEANS OF SOLVING THE PROBLEMS] 

As shown in Fig. 1, the document image re-arrangement 
filing apparatus according to the present invention comprises: 
document image input means 1 for inputting by scanning document 
image such as a scrap of a newspaper; region dividing means 2 

25 for automatically extracting from the document image input by 
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the document image input means 1 different attribute regions 
such as a headline region (3-1) , a text region (3-2) , a drawing 
region (3-3), a photograph region (3-4) and the like, and dividing 
them; itemized file storing means 3 for accumulating and storing 
5 items such as the headline, the text region, the drawing region, 
and the photograph region, extracted and divided by the region 
dividing means 2; retrieving means (6, 7) for retrieving data 
of the items stored in the itemized file storing means 3, by 
item; document image re-arrangement means 8 for retrieving 
10 desired document image. by retrieval of the retrieving means (6,. 
■ 7) , and re-arranging the data in the items to output the entire 
document image.. 



15 or the like with use' of a .document image input means ! is divided, 
into each attribute region such as a headline 3-1, a text 3-2, 
a drawing 3-3, and a photograph 3-4 by a region dividing means 
2, and then positional data of the rectangular regions enclosing 
the respective regions is then extracted: coordinates (x, y) 

20 of an upper left corner of the rectangular region; a width w; 
and a height h. At this time, it is determined whether the 
document is written vertically or horizontally. Subsequently, 
the attribute regions the headline 3-1 and the text 3-2 are 
subjected to the character recognition process to be converted 

25 into code data, and the attribute regions the drawing 3-3 and 



[EFFECT] 



The input document image data input by scanning a newspaper 
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the photograph 3-4 are subjected to the image compressingprocess, 
and the image data is filed into the corresponding items. The 
retrieving is executed by either a retrieving means 6 or a 
retrieving means 7 : keyword retrieving (substring matching for 
registered portion key) executed for the headline 3-1 or the 
text 3-2 by retrieving means 6; and analogous image retrieving - 
executed for the drawing 3^3 and the photograph 3-4 by retrieving 
means 7. During retrieving, only one of the image data among 
the headline 3-1, : the text'3-2, the drawing 3-3 and the photograph 
3-4, which is now retrieved is displayed. When the retrieving 
has finished, a re-arrangement processing means 8 executes the 
re-arrangement process for the entire document and displayed. 
The re-arrangement processing means 8 executes repeatedly the 
re-arrangement of the retrieved data to arrange the retrieved- 
data from the upper right towards left in order of. the: headline 
3-1, the text 3-2, the drawing 3-3 and the photograph 3-4, and 
a line is filled with the data, arrange the data to the other 
paragraph ( in the horizontally written document , from the upper 
left toward right) . 

Figs.. 2 and 3 respectively show the division result of 
the image data into the attribute regions and the re-arrangement 
result . 
[EMBODIMENTS] 

The technical idea of the present invention will be 
described below while referring to Fig. 1. 



In this drawing, the document image input means 1 scans 
image document such as a scrap of a newspaper and input the image 
data, and the region dividing means 2 automatically extracts 
the attributes of the image data from the input document image, 
for example, the headline 3-1, the text 3-2, the drawing 3-3 
and the photograph 3-4 . Among the document image extracted with 
each attribute items, the headline 3-1 and the text 3-2 are 
recognized as characters by a character recognition means 4 to 
be converted into character codes, and then stored in an itemized 
file storing means 3 .■■ When the extracted document image is image 
data, the image data compressing means 5 compresses the data 
to be stored in the itemized file storing means 3 . If the stored 
image needs to be displayed, the data can be retrieved with use 
of the 'keyword retrieving means 6 for retrieving character data 
from the headline 3-1 and the text 3-2 as by item arid the analogous 
image retrieving means 7 for retrieving image data from the 
drawing 3-3 and the photograph 3-4. The. document- image 
re-arrangement means 8 displays the final retrieving result 
obtained by these retrieving means. 

The regions can be arbitrarily recognized and divided 
according to the difference of the attributes of the regions; 
area, density, the number of edges, or the like of a text region, 
a headline region, a drawing region, a photograph region and 
the like. 

Fig. 4 is a block diagram showing the constitution example 



of the region dividing means 2. The constitution shown in the 
diagram is based on the technical inventions disclosed in 
Japanese Patent Application No . SHO 63-252920, Japanese Patent 
Application No. SHO 63-291095, and Japanese Patent Application 
No. HE I 1-87039 (inventor of all the applications : Yutaka 
NAKAMURA) by the applicant of the present application, comprises:-, 
binarizing means 201; rectancularized region image storing 
memory 2 02 for storing the rectancularized image; 
rectancularizing- means - . 2.03 for reotancularizing --image; 
binarized image memory 2 0.4 .for storing binarized image; isolated 
point removing means 205 for removing isolated points from the 
binarized image; outline extracting means 206 for extracting 
outline of the image after removing the isolated points; outline 
image memory 207 for storing the extracted outline image; - and* 
image region recognizing means for recognizing image region on: 
the basis of the data stored in the memories 202, 204, and 207. 
The image region recognizingmeans has rectangular region address 
generating section 208, black pixel calculating section 209 and . 
210, text character recognition section 211, rectangular area 
calculating section 212, comparing section 213, area comparator 
(headline/drawing distinguishing section) 214, comparator 
(photograph/character-linear image distinguishing section) 215 
and the like. 

The image is input by a scanner or the like in a multivalued 
form. The input image data is binarized by the binarizing means 



201. The binarized image data is subjected to the 
rectanculari zing process executed by the rectanculari zing means 
203, wherein the black pixels are connected and the connected 
black pixels are enclosed by a rectangular. The obtained 
5 rectangular region is stored in the rectancularized region image 
storing memory 202, and used as an image recognizing unit. 

After the binarizing process, the isolated points of the 
binarized image data are removed by the isolated point removing . 
means 205, and the outline, i.e. , the. edge of the image is then 
10. extracted by the outline extracting means 206, and stored in 
the outline image memory 207. 

The recognition of the region is performed as follows: 
at first, the text character recognition section 211 recognizes 
' • the text area by inferring the character height of the text on 
15 the basis of the distribution of- the heights of the rectangular 
portions in unit of the region enclosed by the rectangular 
obtained by the rectancularizing means 203, and divides the text 
.areas. Next, the headline regions, the drawing regions,- and 
the photograph regions other than the text regions are recognized. 
20 on the basis of the area, the density and the edge of the rectangle . 
That is, the rectangular region address generating section 208 
generates the addresses of the pixels in the rectangular regions 
other than the text region on the basis of the rectancularized 
image data read from the rectancularized region image storing 
25 memory 202. The pixels in the rectangular region are read from 



the binarized image memory 204 and the outline image memory 207 
in accordance with the generated addresses. The number of the 
black pixels in the read pixels are respectively calculated by 
the black pixel calculating sections 209 and 210 . The calculated 
number of the black pixels in the binarized image in the 
rectangular portion is compared with the number of the black 
pixels in the outlined image by the comparing section 213 to 
obtain the ratio thereof. The comparator 215 compares the ratio 
obtained, by the ...comparing section 213 with , a predetermined 
threshold value.. When the ratio is larger than the threshold 
value, the image is determined as a character-linear image region, 
and when the ratio is smaller than the threshold value, the image 
is determined as a photograph region. 

When the- image is determined as a character-linear image '.■ 
region, , the image' is further determined on the basis, of the area ..- 
of the rectangle whether it is the headline region or the drawing 
region. In. order to distinguish the image regions in this way, 
the area of the rectangular region is calculated, by the. <■ 
rectangular area calculating section 212 on the basis of the 
rectancularized region image read from the rectancularized 
region image storing memory 202 . Subsequently, the area 
comparator (headline/drawing distinguishing section) 214 
compares the calculated area of the rectangular region with a 
predetermined threshold value. When the calculated value is 
smaller than the threshold value, the rectangular region is 




determined (recognized) as the headline region, and when the 
calculated value is larger than the threshold value, the 
rectangular region is determined as the drawing region. 

On the basis of the determination (recognition signal), 
5 the image signal of each region is divided and extracted from 
the input image stored in the image memory or the like, and 
subj ected to necessary processes such as character recognition, 
code compression, and addition of the attribute information for 
retrieving, then registered in the itemized file storing section 

10 3 , That is, the scan-in input image is divided into the attribute 
region such as the headline region 3-1, the text region 3-2, 
the drawing region 3-3, and the photograph region 3-4, and the 
positional data of the rectangular regions is then extracted: 
coordinates (x, y) of an upper left corner of the rectangular 

.15 . region; a width w; and a height h . At this time, it is determined* 
whether the document is written vertically or horizontally. 
Subsequently, ■ the headline region and the text region are 
subjected to the character recognition process by the character 
recognition means 4 so as to be converted into code data. The 

20 • drawing region and the photograph region are subjected to the 
image compressing process by the image data compressing means 
5. The itemized file storing means 3 files the processed data 
of the attribute regions by item. 

The registered image in the above-described manner is 

25 performed by the keyword retrieving means 6 in accordance with 

10 



the keyword retrieving (substring matching for registered 
portion key) to retrieve the required image. The retrieval of 
the drawing or photograph is executed by the analogous image 
retrieving means 7. 

An example of the analogous image retrieving executed for 
the drawing 3-3 and the photograph 3-4 will then be described 
in detail with reference to Figs. 5 and 9. The example is based 
on the description disclosed in Japanese Patent Application No . 
'-HE I 1-113.829 (inventor: MAKOTO HIROSE). by the applicant of the 
present application, which, is advantageous in. that the required 
-image data can be retrieved at a high speed. 

Fig. 5 is a block diagram showing the schematic 
constitution of an embodiment the image retrieving apparatus 
capable of performing the above-mentioned analogous image 
retrieving according to the present ' invention . The image 
retrieving apparatus comprises: an image input device 25 used 
by a user or an administrator for inputting image; a display 
device 2 6 for displaying image or the other information to the 
user or the administrator; a calculation processor 27 for 
calculating the degree of the agreement of the images and 
controlling the order of displaying the retrieved image or entire 
apparatus; an image output device 28 for outputting the retrieved 
image in a form of printing; a control instruction input device 
2 9 used by the user to input the tolerance of the image agreement 
determination; and a memory device 30 for storing image and 




information . 

In order to execute the analogous image retrieving 
effectively, the drawing and photograph are processed to add 
the characteristic information in registering in the itemized 
5 file storing section 3. 

The registering operation will be described at first. Fig. 
6 shows an example of the structure of the two-dimensional 
binarized image information sent from the image input device 
•25 . The image information is displayed as image by. the display 

10. device 26. The whole image information is binarized data of 
M x N wherein 0 expresses white pixel, and 1 expresses black 
pixel. The calculation processor 27 prepares character 
information 32 constituted in an architecture shown in Fig. 7 
on the basis of character quantity obtained from the quantization ~ 

15 of the number of value 1 every small region of K x L in the respective : 
regions. The character information 32 and the M x N image 
information 33 are stored in the memory device 30 . In this manner , 
as shown in Fig. 5, a. character information group 34 and the . 
image data group 35 of a plenty of images are. stored in the memory 

20 device 30 . The head image number 31 is an arbitrary number given 
for the calculation processor 27 to recognize a plurality of 
image data. 

The retrieving process of the image data will then be 
described next with reference to the operation process flow chart 
25 shown in Fig. 9. The number 1 of the images desired by users 




is input into the control input device 29 as the tolerance of 
the image agreement determination (1) . 

Upon receiving the image as the retrieving condition, the 
calculation processor 27 prepares character information a K i of 
the image as the retrieving condition, in the same manner as 
the registration (2) . 

After setting j as 1, the value of a register D T to which - 
Ith value is input in order of the degree of disagreement from 
the smaller one is ' set as v K x L (3) . 

The character quantity b JK i. : .of the image number . j is 
calculated (4) . As shown in Fig. 8, In the case where the 
character quantity, is calculated in registering the image and 
registered together with . the image, as the index of the image, 
the registered value is read from the memory device 30. 

The. calculation processor .2*7 calculates the degree of the . 
disagreement Dj between the j th image and the image to be retrieved 
serially from the image number 1 (5) . 

. . In every time when the value Dj is obtained, the value 
Di as the I t h value in order of the degree of the disagreement 
from the smaller one is compared with the value Dj (6). 

When Dj > Di, the desired image is not present in the group 
to the Ith image, and thus a process on the next image is executed. 
At this time, the value j is add with 1(7). 

When the result of the comparison is Dj =s D X/ Di is replaced 
with Dj (7) . The number of images from the image having the 

13 




smallest disagreement value to the I t h image are stored in the 
memory (8) . A process on the next image is then executed and 
the value j is add with 1 (9) . 

Finally, after calculating the degree of the disagreements 
5 of all the stored images, the image having the smaller 
disagreement degree is displayed in the display device 26. 

The calculation of the degree of the disagreements between 
the character information of the images in the character 
information group 34 and the character information of the image 

10 to be retrieved will be described below. The character 
information comprises the character quantity expressed by the 
image number 31 and the matrix having K rows and L columns. The 
character quantity of the image to be retrieved is expressed 
as a K i (k = 1 to K, 1 = 1 to L) , and the character "quantity of 

15 ■ the jth image in the character inf ormation group 14 to be retrieved 
is expressed as b jk L (k = 1 to K, 1 = 1 to L) . Assuming the number 
of 1 in each of the small regions is quantized as binary value, 
the degree of the disagreements between the image to.be retrieved- 
and the j t h image is - obtained by the calculation expressed by 

20 the following formula: 

Dj = 22(a K i0b jlcL ) 
In other words, the total sum of the exclusive OR of the 
corresponding elements is defined as the disagreement. The 
symbol ® denotes the exclusive OR. 
25 Assuming the number of 1 in each of the small regions is 



quantized as multivalued, more specifically, a K i and b jJC L are 
multivalued, the degree of disagreements between the image to 
be a condition and the j t h image is obtained by the calculation 
expressed by the following formula: 
Dj = 22 ■ | (a K i-b jkL ) | 

Prior to the output of the image as the result of the 
retrieval, the re-arrangement of the entire document comprising 
the headline 3-1, the text 3-2, the drawing 3-3 and the photograph 
3-4 is then performed. Figs. 10 and 11 shows the process flow. . 
of the re-arrangement. The processes shown in Figs. 10 and 11 
are connected at (a) and (b) . 

At first, the headline storing position in the itemized 
file storing means 3, which is assigned to the retrieved document 
image is searched to determine whether or not the headline 3-1 - 
is present . (11) . 

When the headline 3-1 is present, it is determined whether 
the headline 3-1 is written vertically or horizontally (12) . 
When the headline 3-1 is written vertically, the. read headline • 
is re-arranged in accordance with the vertical re-arrangement 
procedure (13) . When the headline 3-1 is written horizontally, 
the read headline is re-arranged in accordance with the 
horizontal re-arrangement procedure (14) . 

In spite of the fact that the headline 3-1 is written 
vertically arranged (13) or horizontally arranged (14), the 
headline storing position is further searched to determine 



whether or not the other headline is present (11) . 

When the other headline is not present in the storing means, 
the re-arrangement has finished and the re-arrangement of the 
character lines of the text will be executed next . At this time, 
the next row is set as the upper right end coordinate of the 
character lines (15). 

The storing means is searched to determine whether or not 
the other character line is present (16) . When the character 
line is present, the right edge space present on the right side 
of the character line is compared with a predetermined margin 
provided to the left side (17) . When the left margin is smaller 
than the right space, the re-arrangement of the character line 
is executed (18) The re-arrangement of the character line is 
executed such that the position- subtracted ty the width of a 
character from the right end of the present character line is 
determined as the right end of the next character line (19) . 

When the space on the left side of the character line becomes 
smaller , than the predetermined left margin in the step (17), 
the paragraph is changed and the right end space of the character 
line is set again (20) . The re-arrangement of the character 
line is performed in the same manner as in the step (18) (21) . 

When the other character line to be arranged is not present 
in the storing means in step (16), the re-arrangement of the 
drawing and photograph will be executed next. At this time, 
the upper right end coordinate of the drawing and the photograph 




(22) . 

When the drawing or the photograph is present in the 
document image in step (23) , the right edge space present on 
the right side of the drawing or the photograph is compared with 
5 a predetermined margin provided to the left side (24) . When 
the left margin is smaller than the right space, the 
re-arrangement of the drawing and the photograph is executed 
(25) . The re-arrangement of the drawing and the photograph is 
executed- such that the position subtracted by the width .of the 

10 drawing and the* photograph from the right end of the present 
drawing or the photograph is determined as the right end of the 
next drawing or photograph (26) . 

In the case that the predetermined left margin becomes 
bigger in step (24) ,* the paragraph is changed and the right-end 

15 space of the character line. is set again (27) . There-arrangement 
of the drawing or the photograph is performed in the same manner 
as in the step (25) (28) . When it is determined that no more 
drawing or photograph is. present in the storing means, the 
re-arrangement process has finished (20) . 

20 [ADVANTAGES OF THE INVENTION] 

According to the present invention, the different 
attribute regions such, as headline region, text region, drawing 
region, photograph region or the like in a document image can 
be automatically extracted from the document image, divided and 

25 classifiedby item (attribute region) tostorethem. Inaddition, 




in retrieving the stored image data, the image data can be 
retrieved in accordance with the process suitable for the 
attribute region to be retrieved. Further, during the retrieval, 
only the attribute region now retrieved can be displayed. With 
5 such a retrieving method, the retrieval can be performed so 
effectively. 

Further, the retrieved document image is re-arranged by 
classifying and collecting them by item, the redundant space 
in the document .can be removed and data can be compressed, and • 
:10 the period of time required for retrieving can be. reduced. 
4. BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram showing the constitution of the 
document image re-arrangement filing apparatus according to the 
present invention; Figs. 2 and 3 show the modes of filing the 
15 . respective attribute regions abstracted by the region dividing; ■ 
Fig. 4 is a block diagram showing the constitution of the region 
. dividingmeans; Fig. 5 is a block diagram showing the constitution 
. of the image retrieving apparatus; Fig. 6 shows the structure 
of the two-dimensional binarized image data; Fig. 7 shows the 
20 architecture of the character information; Fig. 8 shows the 
character information group and the image information group; 
Fig. 9 is a flow chart of the retrieving process; Figs. .10 and 
11 are flow charts of the re-arrangement process by the 
re-arrangement means 8 shown in Fig.l. 

25 




1 ... document image input means 

2 ... region dividing means 

3 ... itemized file storing means 

4 ... character recognition means 
5 5 ... image data compressing means 

6 ... keyword retrieving means 

7 ... analogous image retrieving means 

8 ... document image re-arrangement means 
. . Applicant FUJI XEROX CO. LTD. 

10 Attorney SHOUICHI IWAKAMI 
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